Adopting A Particle Swarm-Based Test Generator Strategy For Variable-Strength And T-Way Testing

Recently, researchers have started to explore the use of Artificial Intelligence (AI)-based algorithms as t-way (where t indicates the interaction strength) and variable-strength testing strategies. Many AI-based strategies have been developed, such as Ant Colony, Simulated Annealing, Genetic Algorithm, and Tabu Search. Although useful, most existing AI-based strategies adopt complex search processes and require heavy computations. For this reason, existing AI-based strategies have been confined to small interaction strengths (i.e., t≤3) and small test configurations. Recent studies demonstrate the need to go up to t=6 in order to capture most faults. This thesis presents the design and implementation of a new interaction test generation strategy, known as the Particle Swarm-based Test Generator (PSTG), for generating t-way and variable-strength test suites. Unlike other existing AI-based strategies, the lightweight computation of the particle swarm search process enables PSTG to support high interaction strengths of up to t=6. The performance of PSTG is evaluated using several sets of benchmark experiments. Comparatively, PSTG consistently outperforms its AI counterparts and other existing strategies as far as the size of the test suite is concerned. Furthermore, the case study demonstrates the usefulness of PSTG for detecting faulty interactions of the input components

A Domain-Region Based Evaluation of ML Performance Robustness to Covariate Shift

Most machine learning methods assume that the input data distribution is the same in the training and testing phases. However, in practice, this stationarity is usually not met and the distribution of inputs differs, leading to unexpected performance of the learned model in deployment. The issue in which the training and test data inputs follow different probability distributions while the input-output relationship remains unchanged is referred to as covariate shift. In this paper, the performance of conventional machine learning models was experimentally evaluated in the presence of covariate shift. Furthermore, a region-based evaluation was performed by decomposing the domain of probability density function of the input data to assess the classifier's performance per domain region. Distributional changes were simulated in a two-dimensional classification problem. Subsequently, a higher four-dimensional experiments were conducted. Based on the experimental analysis, the Random Forests algorithm is the most robust classifier in the two-dimensional case, showing the lowest degradation rate for accuracy and F1-score metrics, with a range between 0.1% and 2.08%. Moreover, the results reveal that in higher-dimensional experiments, the performance of the models is predominantly influenced by the complexity of the classification function, leading to degradation rates exceeding 25% in most cases. It is also concluded that the models exhibit high bias towards the region with high density in the input space domain of the training samples

IoT Anomaly Detection Methods and Applications: A Survey

Ongoing research on anomaly detection for the Internet of Things (IoT) is a rapidly expanding field. This growth necessitates an examination of application trends and current gaps. The vast majority of those publications are in areas such as network and infrastructure security, sensor monitoring, smart home, and smart city applications and are extending into even more sectors. Recent advancements in the field have increased the necessity to study the many IoT anomaly detection applications. This paper begins with a summary of the detection methods and applications, accompanied by a discussion of the categorization of IoT anomaly detection algorithms. We then discuss the current publications to identify distinct application domains, examining papers chosen based on our search criteria. The survey considers 64 papers among recent publications published between January 2019 and July 2021. In recent publications, we observed a shortage of IoT anomaly detection methodologies, for example, when dealing with the integration of systems with various sensors, data and concept drifts, and data augmentation where there is a shortage of Ground Truth data. Finally, we discuss the present such challenges and offer new perspectives where further research is required.Comment: 22 page